"Multifunctional integrated visual system with CMOS or 

CCD technology matrix" 

Background of the invention 

The present invention relates to a visual system, 
in particular for use on motor cars, for detecting en- 
vironmental parameters, such as a misting of the motor 
vehicle windscreen, or the presence of rain drops on 
the windscreen, or low lighting conditions when going 
through a tunnel, under a bridge or due to dusk, or the 
presence of mist and fog, or the meeting with another 
vehicle, said system to be used also for monitoring the 
scene before the vehicle ("front monitoring"), in order 
to detect for instance the presence of a bend with such 
a time anticipation to cause the movement of the adap- 
tive headlight before the bend begins, or to signal a 
side movement of the vehicle towards lane-marking lines 
("lane warning") . 

Summary of the invention 

The aim of the present invention is to carry out a 
relatively simple and reliable system that can effi- 
ciently perform all or part of the aforesaid functions. 
A further aim is to overcome current systems for motor 
vehicles providing for package integration of different 
sensors . 

In the light of achieving said aim, the object of 
the invention is a multifunctional visual system com- 
prising a CMOS (or CCD) technology visual matrix having 
a sensitive area divided into sub-areas each designed 
for one or more functions in the monitoring of the 
scene or in the detection of environmental parameters. 
The division into sub-areas takes place through inte- 
. gration of several optical "imaging" systems and other 
optical systems. 

The invention aims in particular at integrating 
into a motor vehicle a visual system of the type re- 



f erred to above, using a monochromatic, linear or loga- 
rithmic VGA CMOS matrix, to be placed for instance 
close to the inner rear- view mirror of the motor vehi- 
cle, so as to perform several functions among: rain de- 
tection, misting detection, fog detection, dusk detec- 
tion, detection of driving in a tunnel, detection of 
meeting with another vehicle, front monitoring. 

The detection of some environmental parameters, 
such as fog and rain, can be carried out both with an 
active technique, i.e. with an emitter, as shall be 
disclosed in further detail in the following, and with 
a passive technique, i.e. without emitter. 

In first simpler embodiment of the visual system 
with CMOS matrix according to the invention, said ma- 
trix has its sensitive area divided into specific sub- 
areas designed for front monitoring, for passive fog 
detection, for dusk detection, for detection of driving 
in a tunnel and for active fog detection, respectively. 

In a second more complex embodiment, the sensitive 
area of the matrix also includes a specific sub-area 
for rain and misting detection. 

In a still more complex third solution, the sensi- 
tive area of the matrix also comprises another specific 
sub-area for detecting the meeting with another vehi- 
cle . 

Still according to a preferred feature of the in- 
vention, the system is equipped with a sub-area dedi- 
cated to an active rain detection by means of an emit- 
ter. Preferably, said area dedicated to rain function 
is also dedicated to windscreen misting function, al- 
ways by means of an emitter. 

Still according to a further preferred feature, 
dusk function is performed by a specific sub-area of 
the CMOS matrix. Tunnel function is performed by using 
part of the area dedicated to front monitoring func- 
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tion. Fog function is performed both through a dedi- 
cated sub-area with an active technique (i.e. via an 
emitter, for instance a LED or laser diode) and with a 
passive technique in another sub-area, the latter being 
5 included in the one dedicated to front monitoring. 

Vehicle meeting function is performed by using two 
dedicated sub-areas, each having its own system of 
lenses and filters, or a sub-area dedicated to front 
monitoring, by means of optical filters laid with a 
10 discretization degree at pixel level. 

Still according to the invention, it is also pro- 
vided that the sensor has a protection window made of 
glass or transparent plastic material, which also acts 
as support for optical fibers and, if necessary, a 
15 prism; these optical components are fitted into holes 
made into said window, whose function shall become ap- 
parent in the following. 

Still according to a further feature of the inven- 
tion, it is provided for an optical insulation system 
2 0 between the area dedicated to front monitoring and 
those dedicated to rain, misting, fog and dusk func- 
tions, based on a partial covering of the surface of 
the matrix protection window, on the side towards the 
matrix, with a layer made of absorbing or reflecting 
25 material, for instance through serigraphy on thermal 
evaporation . 

Still according to a further feat ure of the inven- 
tion, it is provided for an optical insulation system 
of the area dedicated to rain function from the influ- 

30 ence of the objective for front monitoring function, 
which insulation is based on a partial covering of some 
faces of the prism with a layer made of absorbing or 
reflecting material, for instance through serigraphy or 
thermal evaporation . 

35 As far as the sub-area dedicated to rain function 
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is concerned, the sensor is associated in series with a 
prism provided with optical insulation, an optical 
band-pass filter, an objective oriented perpendicularly 
to vehicle windscreen. 
5 As far as windscreen misting function is con- 

cerned, which is designed to enable the automatic start 
of demisting means the vehicle is equipped with, the 
corresponding sub-area is associated in series with a 
prism provided with optical insulation, a filter and an 
10 objective oriented perpendicularly to vehicle wind- 
screen. 

For dusk function it is provided for an optical 
fiber made of plastic or glass for collecting light, 
oriented perpendicularly to vehicle windscreen or with 

15 a deviation of some degrees from this position. 

For tunnel function it is provided for an objec- 
tive, oriented towards the front scene, with a focal 
length suitable for front monitoring function. 

For fog function based on an active technique it 

20 is provided for a "ball" or "grin" lens, or also no- 
lens at all, together with an optical fiber, if neces- 
sary with another "grin" or micro-optical lens, or also 
with no lens at all, together with a high- 
pass/interf erential filter, with a collection lens 

25 whose optical axis is orthogonal to vehicle windscreen 
or front scene. For fog function based on passive tech- 
nique it is provided for an objective oriented towards 
the scene before the vehicle, with a focal length suit- 
able for front monitoring function. 

3 0 For vehicle meeting function it is provided for an 

optical filter (high-pass filter >500 nm or low-pass 
filter <500 nm) together with an objective for each of 
the two sub-areas. 

For front monitoring function it is provided for 

35 an objective with suitable focal length, whose optical 



axis is inclined of some degrees with respect to the 
horizontal plane, oriented in motion direction and at a 
given distance from matrix center. 

Obviously, output signals from matrix visual sys- 
tem are directed towards an electronic system acquiring 
images concerning the various sub-areas (through 
"windowing" in the case of CMOS cameras) and then proc- 
essing them. 

Brief description of the drawings 

Further characteristics and advantages of the in- 
vention shall be evident from the following description 
referring to the accompanying drawings, provided as a 
mere non- limiting example, in which: 

Figure 1 is a schematic view of a first embodiment 
of the matrix sensor according to the invention imple- 
menting all functions discussed above, 

Figures 2, 3 are schematic views showing the oper- 
ating principles of the sensor detecting windscreen 
misting, 

Figure 4 is a schematic view referring to the con- 
figuration of the sensor for dusk/tunnel function; 

Figure 5 is a general diagram showing active fog 
detection, 

Figures 6, 7 show a second and a third embodiment 
of the system according to the invention, implementing 
a smaller number of functions, 

Figures 7A, 7B show a schematic sectioned view and 
a plan view of the sensor belonging to the system ac- 
cording to the invention, 

Figure 8 shows a perspective view of an example of 
embodiment of the system according to the invention, 

Figure 9 shows a variant of Figure 8, 

Figure 10 shows the possible execution variants of 
the optical system for fog detection, 

Figures 11, 12A, 12B show three different perspec- 



tive views of the assembly including the sensor matrix 
according to the invention with the protection window 
associated thereto, and 

Figure 13 is a perspective view of a prism used in 
the system for rain detection according to the inven- 
tion . 

Detailed description of the invention 

Figure 1 of the accompanying drawings shows a pre- 
ferred embodiment concerning the division of the sensi- 
tive area of the CMOS matrix of the sensor according to 
the invention into sub-areas dedicated to one or more 
functions. This embodiment takes into consideration all 
functions mentioned above: front monitoring, vehicle 
meeting, rain, misting, fog (active and passive mode) , 
dusk, tunnel. 

Figure 1 shows the sensitive area of a VGA matrix 
with a hint to the functions performed by each sub-area 
and their features. 

The distribution of the various sub-areas takes 
into consideration some basic criteria: 

1. The inclination of the matrix when mounted onto 
the inner rear-view mirror of the motor vehicle depends 
on the direction of the optical axis for performing 
scene front monitoring function; 

2. For each function the size of the areas is 
function of the fields of view and of the required 
resolution; 

3. The positions of the areas depend on the direc- 
tion of the optical axis of each function and on the 
need for separation areas in which pixels are not used; 

4. A single area can be dedicated to more func- 
tions; or a portion of the area dedicated to one func- 
tion can also be dedicated to another function. 

Possible measuring techniques to be used for de- 
veloping the functions referred to in Figure 1 will now 



be described- 

Windscreen misting 

Passive system : on windscreen outer surface lies a 
reference image, for instance a grid, which is focused 
onto the CMOS matrix. The sharpness level of the image 
depends on the misting degree of windscreen inner sur- 
face. The critical points of this technique are the 
following: the sensitivity to misting levels that can- 
not be seen by human eyes; the dependence of the signal 
on environmental lighting. 

Active system : - as shown in Figures 2, 3, an in- 
frared emitter E sends a bundle onto the inner surface 
of the glass of motor vehicle windscreen P, with an an- 
gle of incidence of about 45°. If the surface is misted 
up, the bundle is partially backscattered by condensate 
droplets (Fig. 3) and detected by CMOS matrix M. If no 
misting is present (Figure 2) , the aforesaid phenomenon 
does not occur. The optical system can shape the bundle 
sent out by emitter E (typically a LED) so that it is 
focused onto a convenient windscreen portion. 

Rain 

The CMOS matrix detects the image of drops lying 
on windscreen outer surface. A quantitative analysis 
can be carried out on a single image, for instance by 
considering the spectrum in space frequency, or by com- 
paring consecutive images with statistical methods. In 
order to eliminate the dependence on outer lighting 
conditions, which affect drop contrast, and to light up 
in a uniform and time-constant way the concerned wind- 
screen area, a LED source in near-infrared radiation 
and a band-pass filter adjusted on LED wavelength are 
used . 

Dusk/tunnel 

Two matrix areas are dedicated to lighting meas- 
ures carried out from two different directions (see 
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Figure 4) : 

- small solid angle A in driving direction (for in- 
stance 10° ) ; 

- larger solid angle B (for instance 40°) oriented up- 
wards, so as to achieve a measure of average lighting 
around current vehicle position. 

Fog 

Passive sensor : passive fog detection is performed by 
scene image acquisition in the area destined to front 
monitoring and subsequent analysis of image sharpness. 
It enables to detect the fog bank in advance with re- 
spect to active system, which has a limited range of 
action . 

Active sensor : the visibility sensor comprises a trans- 
mitting module (LED or laser diode in infrared radia- 
tion) and the receiving module (CMOS camera) . The two 
fields of view partially overlap. In case of fog, the 
droplet concentration in the overlapping area causes a 
backscattering of the bundle, which is detected by the 
sensor (Figure 5) . 

Vehicle meeting 
The CMOS camera frames some portions of the road 
scene before the vehicle. The matrix of Fig. 1 includes 
two "Vehicle meeting" areas respectively dedicated to 
the detection of the headlights of vehicles driving in 
opposite direction and of the rear-lights of vehicles 
driving in the same direction. For the two areas con- 
venient high-pass and low-pass filters for distinguish- 
ing dipped headlights from rear-lights are used. An al- 
ternative consists in using the area designed for front 
monitoring in a color matrix or in a monochromatic ma- 
trix by laying at pixel level the necessary optical 
filters, though only in the matrix area or sub-areas 
designed for front monitoring. 

Scene front monitoring 
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The main portion of the matrix of Fig. 1 is used 
to frame the road scene before the vehicle and to im- 
plement functions such as lane warning, adaptive head- 
light and vehicle meeting. 
5 The embodiment shown in Figure 6 is a simplified 

embodiment with respect to the one in Figure 1, and in- 
cludes only sub-areas for front monitoring, tunnel, fog 
(active and passive technique) and dusk functions. 

Figure 7 shows a further embodiment envisaging in 
10 addition also rain/misting function. 

CMOS matrix 

As far as CMOS visual matrix is concerned, the 
main requirements it should meet for the multifunc- 
tional integration proposed in the previous paragraphs 
15 are the following: 

- VGA format ; this format enables a division of the ma- 
trix into seven used areas and into separation areas in 
which pixels are not used; 

- pixel size : a very small pixel size implies a higher 
20 difficulty in carrying out dedicated optical systems 

for each function/area; 

- monochromatic or color : the use of a color matrix 
would avoid the use of optical filters required for 
"vehicle meeting" function, however to the detriment of 

25 space resolution capability; 

- linear or logarithmic response : some functions are 
based on the quantitative assessment of a parameter for 
which it would be better to have a linear response of 
the sensor and constant integration time; as far as the 

30 dynamic range is concerned, however, it is preferable 
to use a sensor with logarithmic response. 

dynamic range : front monitoring functions require 
that the sensor is able to cover an illumination range 
above 80 dB since typical scenes can involve situation 

3 5 varying from dusk to direct sun rays; 



- sensitivity : this is an important parameter for front 
monitoring functions during night driving with a low 
illumination level (for instance dusk = 1 lux) or for 
fog function when the backscattering radiation col- 
lected is very small (in the range of nW) ; 

- spectral response : the sensor should have a good re- 
sponse in the range 800-900 nm (used for active func- 
tions - where the windscreen glass of a standard motor 
vehicle has a transmittance of about 30%) ; for "dusk" 
function the sensor should have a response as similar 
as possible to the one of human eyes (photometric re- 
sponse) ; 

- frame rate : frame rate is not a critical parameter 
unless fast acquisitions by means of windowing are to 
be made and then digital filtering techniques are then 
to be implemented (for instance in the case of vehicle 
meeting function, in which it would be useful to cut 
the modulated radiation coming from AC-supplied light 
sources with frequencies of 50-60 Hz) . 

In a practical example of embodiment of the inven- 
tion, a logarithmic monochromatic VGA CMOS camera has 
been used, carried out with CMOS technology 0.35 mi- 
cron. It integrates the matrix of active pixels, ampli- 
fication stages, a 10 bit AD converter and the inter- 
face for a microprocessor. The photodiode matrix con- 
tinuously converts radiation into voltage (without 
charge integration as typical of linear CMOS) and pix- 
els can therefore be read at any moment. 

The current generated by the photodiodes has a 
linear dependence on the intensity of the incident ra- 
diation. The supply circuit of the photodiode has a 
logarithmic feature so that pixel voltage output is 
proportional to the logarithm of the incident radia- 
tion. 

With reference to sensor package, the latter is 
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shown in Figure 7A (in lateral section) and in Figure 
7B (showing a view from above) . Said package comprises 
a base 10 onto which the sensor chip 11 is mounted, and 
an optical protection window 12, consisting of a sub- 
5 strate of transparent material, with plane and parallel 
surfaces. The CMOS matrix is associated to optical com- 
ponents designed to optimize the signal collected by 
the integrated sensor for each of the environmental 
monitoring functions; in the case of rain, vehicle 

10 meeting and front monitoring functions, the optical 
components should also carry out a scene imaging. For 
each function the field of view and the dedicated sen- 
sor area determine the focal length of the optical com- 
ponent to interpose. Once the focal length has been 

15 set, the requirements concerning the illumination level 
limit the minimum dimension of the lens. The conditions 
imposed by system geometry should further be taken into 
consideration. As a matter of fact, each of the adja- 
cent areas a matrix is divided into is associated to an 

20 independent optical system. On a design level this kind 
of integration implies other two conditions: 

- the optical systems should be placed one beside the 
other. This limits the size of single components. In 
the evaluation of the limits possible mechanical solu- 

25 tions for mounting and packaging operations should also 
be considered ; 

- the proximity of the areas results in disturbances 
among the signals of the single functions, due to the 
partial overlapping of image planes. Concerning this 

3 0 problems, a separation system for the single areas 

should be designed. 

Another feature to be taken into consideration is 

the orientat ion of the optical axis for every function. 

It is evident that to meet all requirements it is nec- 
35 essary to find solutions for changing the direction of 
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the optical axis. To this purpose the invention, as 
shall be evident in the following, uses prisms and op- 
tical fibers. 

Since for all functions the distance of the target 
spot is over 10 times focal length, the distance opti- 
cal system- sensor (image spot) shall be approximately 
the same as focal length, which for "rain", Musk" and 
"vehicle meeting" (detection of rear-lights) functions 
is below 5 mm. This distance is a strict requirement, 
especially if prisms are to be placed between the ob- 
jective and the sensor. 

Figure 8 shows a perspective view of an embodiment 
of the system according to the invention. The CMOS ma- 
trix 11 equipped with protection window 12 is associ- 
ated to an objective 13 for front monitoring. As was 
already referred to, the objective 13, together with 
the sensor 11, is mounted in the front portion (i.e. in 
driving direction) of the inner rear-view mirror of the 
motor vehicle (not shown for reasons of clarity) . In an 
example of embodiment, the objective 13 has a negative 
lens as first element, so as to cover quite a wide 
field of view (35°-45°) . 

Then there is the problem concerning the insula- 
tion of front monitoring function. A possible solution 
involves the use of a separation wall with rectangular 
opening. The opening has the same shape and size as the 
front monitoring areas. 

Considering now "dusk" and "fog" functions, there 
is the problem concerning the fact that the optical 
axis should be inclined upwards for instance of an an- 
gle of 60° with respect to motion directions. Detection 
is of non- imaging type. 

For these two functions the invention solves the 
problem concerning the inclination of the optical axis 
by using conventional optical fibers, such as the opti- 
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cal fiber 14 in Figure 8. Such an optical fiber enables 
to transmit the signal along a path that can also be 
curved. Said component is further quite cheap. The op- 
tical fiber is preferably associated to a lens, for in- 
5 stance a ball lens or a grin lens (gradient of index) . 

Figure 8 also shows an objective 15 for rain de- 
tection, and an emitter 16 to which a collection lens 
17 and an optical fiber 18 for fog detection are asso- 
ciated . 

10 Figure 9 shows a variant of Figure 8, in which 

corresponding elements have the same reference number. 

Figure 10 shows the various possibility of asso- 
ciation to an optical fiber 18 of a GRIN 19A lens or a 
ball lens 19B, or of leaving the fiber end without lens 

15 as shown in 20. The opposite end of the fiber 18 can be 
provided with a GRIN lens, or a micro-optical system 
22, being it further possible to leave the end without 
lens as shown in 23. The collection lens 17 is associ- 
ated to a band-pass optical filter 24. 

20 A step following the integration of front monitor- 

ing, "fog" and "dusk" functions through objective and 
optical fibers, consists in the additional integration 
of "rain" function. For the latter function imaging op- 
tical components are necessary, i.e. lenses and prisms. 

25 The field of view of front monitoring function has an 
optical axis inclined of some degrees downwards. For 
"dusk", "rain" and "fog" functions the inclination is 
of 60 degrees upwards. Actually, for the latter three 
functions there is a broad tolerance. A variation of 5- 

3 0 6 degrees does not involve any substantial change as 
far as functionality is concerned. A possible optical 
solution is to keep matrix plane orthogonal to the op- 
tical axis of front monitoring function and, by means 
of prisms, to change the direction of the optical axis 

35 for the other three functions. The tolerance range for 
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orientation enables to select a prism deviating the op- 
tical axis of 60° with respect to the perpendicular to 
image plane (matrix plane) . The prism taken into con- 
sideration (Littrow prism) is referred to with 20 in 
Figure 12A and 12B. Said figure also shows the shaded 
area 31 and the end portions of optical fibers 14 and 
18 fitted into holes made into the optical window 12. 

Figure 13 shows a perspective view of the prism 
30. Said prism 30 is made of glass or transparent plas- 
tic, with its face BC covered by a one- or multi-layer 
coating, so as to obtain a surface reflecting towards 
the inside of the prism and absorbing or reflecting to- 
wards the outside. The triangular faces ABD of the 
prism 30 should be covered with a one- or multi-layer 
coating, so as to obtain a surface absorbing towards 
the inside of the prism and absorbing or reflecting to- 
wards the outside. 

The prism 30 should be oriented as in Figure 12, 
with its face AD parallel to sensor plane; the face CD 
should further not rest on any optical element, for 
reasons that shall be evident in the following. Before 
the face AC of the prism 3 0 the imaging objective 15 
should be placed, which should have such a rear focal 
length to be able to focus beyond the prism. The light 
getting out of the objective strikes the face AC and 
gets into the prism 30; if the angle of incidence onto 
AC is smaller than a limit angle (which for prism-air 
passage is of 27.9° with a refractive index of the 
prism of 1.5) , light is reflected wholly onto the face 
CD; this results in the need for air as optical medium 
outside the face CD. Eventually, the bundle gets out of 
the prism 3 0 through CD towards the sensor. 

For the optical insulation of the bundle getting 
out of the prism 30, a rectangular opening having the 
same surface as the section of the radiation bundle 
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getting out of the prism 3 0 or a larger surface, should 
be made into the optical protection window 12. The in- 
ner wall of the opening should be covered with an ab- 
sorbing coating, so as to avoid disturbances with the 
5 signal of front monitoring function. If necessary, all 
inner walls can be covered with said coating in case of 
disturbances with the signals from the optical fibers. 

This kind of optical system (objective plus prism) 
can also be used together with the objective for front 

10 monitoring function. The fact that the two optical sys- 
tems do not have the same axis is advantageous as far 
as size is concerned. 

Obviously, though the basic idea of the invention 
remains the same, construction details and embodiments 

15 can widely vary with respect to what has been described 
and shown by mere way of example, however without leav- 
ing the framework of the present invention. 



